~- International Journal of Designs for Learning 


2016 | Volume 7, Issue 2 | Pages 41-56 


DESIGNING FOR CONSISTENT IMPLEMENTATION OF A 5" GRADE 
DIGITAL MATH CURRICULUM 


Jeremy Roschelle', Steven Gaudino?, & Samantha Darling? 
'SRI International; 7Reasoning Mind 


Reasoning Mind products are used by over 100,000 students 
a year and have shown positive outcomes. In this design 
case we focus on implementation: how Reasoning Mind's 
approach evolved to tackle the challenge of achieving 
consistently high-quality implementations with many 
different schools, teachers, and students. Key insights include 
the definition of the Implementation Coordinator role and 
how that role is managed, the design and refinement of 
specific tools to support the implementation improvement 
process, and how Reasoning Mind's understanding of its 
organizational values in relationship to its approach to imple- 
mentation evolved over time. Based on a study in which 23 
schools in West Virginia are newly adopting Reasoning Mind, 
we also reflect on how the design insights are playing out in 
a large-scale implementation. 
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INTRODUCTION 


Improving mathematics learning is a long-standing goal 

for the use of technology in education. Innovators have 
developed novel approaches, such as the use of intelligent 
tutoring systems to guide students’ learning. More recently, 
digital curriculum products are replacing conventional paper 
textbooks as the major classroom resource that guides 
instruction in mathematics throughout the year. Although 
many innovative approaches achieve positive results when 
evaluated in just a few classrooms, variability in how the 
approach is used by teachers and schools can result in mixed 
results in larger evaluations. 


For example, a prominent experiment evaluated the most 
promising products available at the time and found that the 
impacts of digital mathematics products on student learning 
averaged out to be close to zero (Dynarski et al., 2007). In 
some classrooms in this study, student outcomes increased; 
in others, student outcomes decreased. Given the variability, 
a positive overall effect was not detected. 


Even when the research basis for an innovative approach is 
exceptionally strong, implementation issues can interfere 
with student learning impacts. For example, Cognitive 
Tutor Geometry has a strong research-backed design, but 
one large scale trial found that students assigned to use 
the tutor scored 19% of a standard deviation lower than 
students in control classrooms (Pane, McCaffrey, Slaughter, 
Steele, & Ikemoto, 2010). Researchers attributed the drop to 
implementation stating: "data suggest that many teachers 
had difficulty implementing the treatment curriculum’s 
learner-centered pedagogy” (Pane et al., 2010, p. 254). The 
challenge of achieving a consistent implementation has 
also been noted for the Khan Academy (Murphy, Gallager, 
Krumm, Mislevy, & Hafter, 2014) and other newer designs for 
blended learning (Murphy, Snow et al., 2014). 


Conversely, when evaluations of technologies for mathe- 
matics learning have found stable positive effects, attention 
to the quality of implementation has often been a notable 
design feature. Technology-enhanced, Research-based, 
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Instruction, Assessment, and professional Development 
(TRIAD), a digital mathematics curriculum for early mathe- 
matics learning, has strong evidence of efficacy and features 
a strong professional development component (Clements 

& Sarama, 2011). An evaluation of networked graphing 
calculators found positive effects on students’ algebra 
learning and noted the extensive focus in the implementa- 
tion on improving teachers’ formative assessment practices 
(Pape et al., 2012). In another evaluation that found positive 
effects on mathematics learning, the approach to learning 
mathematics featured a "curricular activity system” approach 
which defines and supports the alignment and integra- 

tion of curriculum, technology, and teacher professional 
development (Roschelle et al., 2010). Overall, there is now 
broad recognition that designing the implementation of a 
technology is a distinct phase of design from designing how 
the technology directly supports mathematics learning, and 
that specific design processes are appropriate to creating 
powerful supports for consistently high quality implementa- 
tion (Penuel, Fishman, Cheng, & Sabelli, 2011). 


n this design case, we focus on how Reasoning Mind's ap- 


plays in mediating the outcomes that SRI seeks to measure 
in its studies, SRI now designs its studies with considerable 
attention to (a) allowing time to adaptively improve imple- 
mentation before outcomes are measured and (b) measur- 
ing implementation through multiple instruments, including 
interviews, observations, teacher logs, and system data. 


The second author, Steven Gaudino joined Reasoning Mind 
as a math curriculum developer in 2004. In 2005-2006, he 
recruited and led a team of editors and testers in ensuring 
the correctness and proper functioning of online education- 
al content. Also in this period, he organized and ran one of 


Reasoning Mind's first pilo 


t projects, using the curriculum he 


had helped develop. In 2007, Gaudino became Reasoning 


oversaw a massive expans 
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THREE PERSPECTIVES ON IMPLEMENTATION 


The three authors of his article are working together on the 
West Virginia study. 


The lead author, Jeremy Roschelle, has been involved in 
many evaluation studies of digital mathematics products 
that have been conducted by SRI over the past decade. 
Across these studies, he has seen many implementation 
issues that occur across products. Sometimes products 

are purchased by schools but are not used in classrooms 

as intended, or are used too little to make a difference. 
Sometimes, the product is used in classrooms, but the 
teachers’ role is unclear or unsupported, and the teachers’ 
contribution to student learning declines. A key challenge to 
successful "blended" learning approaches is balancing and 
integrating the roles of teachers and technology in support- 
ing student learning. SRI has also seen digital mathematics 


who was energized when she learned how Reasoning Mind 
taught mathematics. She joined Reasoning Mind to take 
on the newly defined Implementation Coordinator role; 
an Implementation Coordinator works with schools and 
teachers throughout initial training and during the school 
year to support their achievement of high quality implemen- 
tations. Now Darling oversees a team of Implementation 
Coordinators. Her perspective is grounded in the realities 
of working with teachers and schools on a daily basis to 
resolve implementation challenges; she appreciates the role 
of standardized processes and supports in this respect, and 
also the ways in which implementation improvement must 
build on a foundation of strong mentoring relationships with 
teachers. 


From the perspective of Reasoning Mind, the design lessons 
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perspectives are marked with a "@". 


ABOUT REASONING MIND 


The focus of the design case that will follow is on the design 
of the support for implementation of Reasoning Mind's 5° 
grade core curriculum—much of which relates to how 


42 


teachers are supported as they use Reasoning Mind with 
students through a school year. Design cases could also be 
prepared on other layers of design that are important to the 
system, such as the curricular and pedagogical design and 
the user experience design. Each of these design layers has 
its own criteria and processes, and its own design history. 
We begin with some background details on other design 
layers below, but focus our design case thereafter on the 
implementation design. 


Background: Instructional Program 


Reasoning Mind, a nonprofit organization, offers digital 
mathematics products that can be used either as a supple- 
ment to conventional instruction or as the core instructional 
materials for a grade level. We focus on Reasoning Mind's 
core instructional program for 5" grade, as this is what we 
are studying in West Virginia. In addition, implementing a 
new core curriculum is more intensive for teachers than 
implementing a supplement. 


The Reasoning Mind core program is a complete system 
for teaching 5" grade mathematics, intended to be used by 
teachers and students for 70-90 minutes a day. This sched- 
uled time was chosen to ensure that students complete 
their assigned curriculum over the course of the year with 
students completing a lesson, on average, each day. The 
amount of time is reported here because it is contextually 
important to understanding the implementation design; 
when teachers use a product as a core curriculum (rather 
than as an optional supplement), implementation issues 
become more important to address. 


na classroom using this system, students begin working 
individually on mathematics at a computer when they 

arrive in class and continue working individually for most of 
the time. The teacher also has a computer, which provides 
control over what is going on in class as well as reports fea- 
turing data about students’ progress. The teacher sometimes 
addresses the whole classroom for a short period of time to 
discuss mathematics, logistics, or issues of motivation (e.g., 
goals, incentives, celebrating successes). The teacher spends 
substantial time working with individual students or with 
small groups of students; sometimes students ask for sup- 
port from the teacher and, at other times, the teacher can 
use data to decide which students to work with. In addition, 
the teacher can use data to orchestrate peer mentoring and 
collaboration by matching students on similar topics or by 
pairing advanced students with those in need of additional 
support. This can be done through intentional peer seating 
or by designating students to serve as teacher assistants on 
a given day. 


Thus, the Reasoning Mind core program is a blended learn- 
ing approach in which students’ experience is of learning 
both from the technology and from interactions with their 
teacher and peers— and the technology and the teacher 
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are helping the students with the same mathematical 
challenges. 


Background: Curriculum Design 


The Reasoning Mind curricular design was built through 
Instruction Modeling, the process of examining effective in- 
struction and recreating it through technology. Khachatryan 
et al. (2014) provide a design case for the curricular design, 
which we summarize as follows: By studying the methods 
of expert Russian teachers in their traditional classroom 
settings, Reasoning Mind found that quality mathematics 
instruction relies on a range of factors including: (a) the co- 
herence of the written curriculum, (b) the activities expected 
of students, (c) the adaptations of the teacher, (d) the culture 
of the classroom, and (e) the nature of interactions between 
teachers and students. To capture these practices, Reasoning 
ind’s team of expert Russian teachers provided detailed 
scripts that described the moment-by-moment actions of 
teachers and students in each lesson. Each script was then 
studied and elaborated upon through extensive interviews. 
The interviews targeted the motivation behind each teacher 
decision. 


Once this was fully understood, Reasoning Mind's 
Knowledge Engineers developed a novel technology-based 
user experience that recreated the set of experiences a 
student would receive in the expert teacher's classroom. 

For example, a key curriculum principle is that students 
learn through visual models, such as using an area model 
for multiplication (see Figure 1). Many design decisions 

in Reasoning Mind follow in a straightforward way from 

an analysis of existing high quality curricula and teaching 
practices. Further, the way these decisions were made 
would be unlikely to generalize to most learning technology 
products, few of which are made by observing teaching in 
one culture and seeking to re-produce that teaching process 
in another culture. As such, these decisions are not the focus 
of our design case. 


Background: User Experience 


In a Reasoning Mind classroom, each student logs onto 

an individual account that continuously adjusts to the 
student's pace and level of mastery as he or she study the 
curriculum. Unlike technology programs that break from 
traditional instruction, the Reasoning Mind system uses an 
intelligent tutoring system to present and adapt in the ways 
expert teachers would. Unlike curricula divorced from their 
cultural context, the Reasoning Mind system attempts to 
preserve the pedagogical methods that originated along 
with its curriculum’s tradition. The design intention in the 
user experience is to provide students with a consistent 
quality of instruction that maintains fidelity to the curriculum 
and the proper method of teaching it. The design approach 
builds on prior research in intelligent tutoring systems. For 
readers interested in the design issues relating to authoring 
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FIGURE 1. The Reasoning Mind curriculum takes advantage of international 
pedagogical practices such as using problems to drive content knowledge 
and integrating geometry to illustrate arithmetic concepts. In the screen 
shot above, area is used to motivate fraction multiplication. Students then 
develop an abstracted algorithm for this process through a series of guided 
questions. Based on each student response, the system provides tailored 
feedback and adjusts the content it delivers. 


intelligent tutor systems more generally, a longstanding 
literature already exists (e.g., Murray, 1999). We focus on 
design of the implementation, for which not much literature 
is available. 


Reasoning Mind also goes beyond the classic intelligent 
tutoring system by supporting peer learning in the class- 
room. As previously mentioned, the teacher can group 
students to work together in pairs or designate a student 

as a teaching assistant for the day. Also, teachers can use 
data reports to pair struggling students with those who 
demonstrated fluency. In addition, students are encouraged 
to work together on challenging problems in the Wall of 
Mastery. As with intelligent tutoring systems, there is already 
an extensive existing literature to guide design of collabora- 
tive learning (e.g., Kirschner & Erkens, 2013) for readers with 
further interest in the issues of designing for collaborative 
learning. 


Background: Role of Teachers 


Teachers need preparation, training, and support to use 
Reasoning Mind, as is the case with any core curriculum 
product. A new teacher's experience of Reasoning Mind 
begins with a "Qualification Course,’ which takes two to three 
days over the summer. The Qualification Course covers the 
things a new Reasoning Mind teacher needs to know to suc- 
cessfully launch the program with their students. Highlights 
include training on the student and teacher interfaces, 
data-driven intervention techniques, hands-on curriculum 
exploration, and best practices for managing a successful 
blended learning classroom. After the Qualification Course, 
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teachers participate in ongoing professional 
development (PD) throughout the school year, 
which includes both PD modules and personal- 
ized coaching. 


The Reasoning Mind team believes that a strong 
quality of student learning will not occur without 
teachers who lead a high quality classroom 
implementation. A classroom environment that 
is not conducive to learning will dismantle any 
instructional approach. But even with strong 
classroom management, technology is not 
powerful enough to model all key practices, so 
the content must be supported by additional 
system components and coordinated teacher 
instruction. The practices that require special 
implementation focus include: (a) student 
mindset, (b) student practices, (c) classroom 
procedures, (d) instructional prioritization, and 
(e) conceptual discussion in small group settings. 
Reasoning Mind fosters these practices through 
its virtual mentorship, student interface, data 
tools for partners, and Implementation support 
as we describe next. 


REASONING MIND FEATURES AND 
IMPLEMENTATION NEEDS 


Building on a system that has previously established designs 
for its instructional, curriculum, user experience, and teacher 
components, Reasoning Mind addresses implementation 
holistically, as a new layer of design and not as a set of inde- 
pendent issues. Describing three important aspects of the 
Reasoning Mind approach is useful in clarifying how teacher 
implementation of Reasoning Mind matters to students’ 
experience and learning. 


The Genie Character 


An animated character called The Genie guides the student 
through each lesson (see Figure 2). The Genie's primary func- 
tion is to foster a productive mindset through its immediate 
feedback, solutions to problems, and words of encourage- 
ment. This consistent, positive presence can help students 
stay focused and persevere through challenging material. 


To develop the student's connection to the Genie, Reasoning 
Mind employs a team of writers who respond to hundreds 
of student emails daily. For consistency, each writer responds 
according to a carefully constructed biography. Components 
of the Genie, such as its neutral gender and lack of voice, 
were intentionally designed to ensure that the character was 
equally relatable to all students. 


Over time, Reasoning Mind refined the Genie’s actions 
based on student and teacher feedback. Its reward response 
movements became more elaborate and varied, its emails 
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FIGURE 2. The Genie character guides students through each 
lesson. 


to students drew from research on student mindset, and its 
solutions to incorrect problems became more interactive. 


Though students respond positively to Genie, the teacher 
still sets the tone of the Reasoning Mind classroom, and stu- 
dent attitudes are modulated by the teacher's own mindset. 
This impact on the student experience means that teachers 
must have the tools and training needed for the program to 
have its intended effect. 


Notebooks and Note Taking 


Paper notebooks are also a prominent feature of Reasoning 
Mind's classrooms. Students are required to have notebooks 
and are guided to write specific mathematical definitions, 
procedures, and problem solutions in their notebooks. To 
develop productive note taking habits, the system labels 
key concepts with a "Write this Down’ tag. At the end of 
the lesson, students are given a Notes Tests that serves as a 
review of the key concepts and a check of whether students 
recorded them in their notebooks. Students note taking 
habits are expected to develop throughout the year. 


The purpose for the notebooks is to develop students’ 
independent learning strategies—skills that enable students 
to learn mathematics effectively on their own. Teachers are 
critical to setting norms for notebook use, but also in guiding 
students to use their notebooks as a learning resource. For 
example, teachers help students to understand that they 
will have to keep notebooks that are clear and neat enough 
to be useful later. Further, teachers direct students to use 
notebooks as a primary learning resource when they are 
stuck. For example, many teachers implement a "3 before 
me” routine, where students must try to resolve their own 
challenge using their notebook, online resources, or a 

peer before asking the teacher for help. The importance of 
notebooks and of teachers’ role in supporting note taking 
practice need to be reinforced throughout the training and 
coaching for Reasoning Mind teachers. 
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Data-Driven Decisions 


Reasoning Mind collects data as each student solves 
mathematics problems and progresses through the system. 
This data is used by the system’s algorithms to adapt to the 
students’ needs—for example, a student who is achieving 
low accuracy on a set of problems may be directed to a 
diagnostic module and to additional learning resources for 
that topic. In addition, a key feature of Reasoning Mind's 
instructional module is that teachers are expected to use 
data to make classroom decisions. 


The earliest reports provided insight into student perfor- 
mance, but they lacked the level of detail needed to pull 
small groups and provide intervention. For example, an 
overall percentage of problems solved correctly allowed 
teachers to know if students were being successful, but if 
the percentage was low, the teacher could not conveniently 
identify the particular areas of confusion. Through partner 
feedback, Reasoning Mind developed a set of features to 
support teachers in making instructional decisions. Teachers 
now can use an objective spreadsheet report to sort students 
by performance over any topic in order to form small 
groups for intervention or pair students for peer tutoring. To 
intervene, teachers need to see what problems the student 
got wrong and what incorrect solution was entered. The 
teacher can access this information for a particular student 
on a particular objective by clicking on the corresponding 
cell that shows the accuracy percentage. This opens the stu- 
dent's activity logs that contain the problems the student got 
wrong and what the student's answer was. The new dataset 
button generates similar problems to solve, so teachers can 
see a student's reasoning first-hand and then discuss the full 
solution together. 


Teachers also get status notifications (see Figure 3). These 
notifications provide updates on student work in real time, 
which can give the teacher insight into whether students 
are spending their time productively. Learning these features 
takes time, and when teachers begin with Reasoning Mind, 
they typically have not previously used data as intensively to 
make real-time classroom decisions about which students to 
work with and what to work on. Thus an important concern 
for implementation fidelity is to help teachers incorporate 
Reasoning Mind's capabilities for data-driven decisions 
into the core of their teaching repertoire and daily practice. 
Similarly, the Genie and Note taking features can only realize 
their design intent though teacher support. As teachers be- 
come skilled on what to do incrementally, implementation 
support for teachers is critical to Reasoning Mind's approach. 


RESEARCH AND REASONING MIND 


Reasoning Mind has engaged external researchers in its work 
since its inception. SRI reviewed a series of prior research 
studies conducted by other investigators and found that 
effects on student learning were consistently positive 
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Today, May 2, 2016, cont’d. 


09:18 AM Bryan Rowey’s accuracy on the objective Review: Adding and Subtracting Decimals dropped below 60% 


09:18 AM Ethan Deleon solved 20 problems in a row correctly 

09:17 AM Leonardo Bravo solved a C-level problem correctly 

09:14 AM Felicity Lanham solved 20 problems in a row correctly 
09:14 AM Christian Madden solved 5 problems in a row correctly 
09:13 AM Chigozirim Osuji started the objective Comparing Decimals 
09:12 AM Kathleen Daniels solved 10 problems in a row correctly 
09:12 AM Prajesh Ghalley solved 20 problems in a row correctly 
09:10 AM Felicity Lanham started the objective Comparing Decimals 
09:10 AM Aliyah Gulley solved a C-level problem correctly 


Customize notification settings 


02 May 2016 


Tutoring Others 


Achievements (Scr groT-s 


( Working after school time 

(Y Working on weekend 
(Reached D6 in Review Mode 
(Level C Solved 
(Y5(10,15,etc.)-in-a-row correct 
(Highest streak in class 

(V Highest personal streak 
(Passed Objective 

(Previously Failed Module Passed 
(2 Diagnosis Remedied 


(2 100% on homework 
{V4 100% on Notes Test 


FIGURE 3. Teachers can customize their notifications. 


(Roschelle, Bhanot, Patton, & Gallagher, 2015). Teacher and 
student attitudes, as reported by external evaluators, show 
high engagement and motivation. SRI's report also noted 
that Reasoning Mind has shown an ability to achieve a high 
quality implementation at scale, for example, with over 
25,000 students in Dallas. 


SRI is presently conducting a study of Reasoning Mind in 
West Virginia with funding from the U.S. Department of 
Education's Institute of Education Sciences. The study is de- 
signed to evaluate the impacts of Reasoning Mind's 5“ grade 
core curriculum for mathematics on the state's end of year 
mathematics test, compared to impacts of business-as-usual 
approaches to teaching the same mathematics. Over 50 
volunteer schools were recruited and randomly assigned to 
use Reasoning Mind either as a 5" grade core curriculum 
(the treatment condition) or to continue with their existing 
curriculum for 5‘ grade (the control condition). To date, the 
research team has had the opportunity to observe class- 
rooms and survey teachers, and has been able to inspect 
the system log data that result from teachers’ and students’ 
use of Reasoning Mind. After describing Reasoning Mind's 
designs to support implementation, we will return to this 
study to reflect on how our observations in the first year of 
implementation in West Virginia relate to the designs, thus 
enriching the design case with observations from a large- 
scale implementation. Collecting and analyzing this data is 


JDL | 2016 | Volume 7, Issue 2 | Pages 41-56 


(Finished the assigned curriculum 
(Higher than 90% on Standardized Test Practice 
(2 Won Checkmark Game in Round 1 


(2 Mastering block in Wall of Mastery 


also instrumental to SRI’s role in com- 
municating insights about how to 
address large-scale implementation 
challenges with broader audiences. 


WHY AND HOW 
REASONING 
MIND TACKLED 
IMPLEMENTATION 


A need for tackling implementation 
as its own design issue emerged 

as Reasoning Mind tested its early 
approach in classrooms. The first 
classroom test of Reasoning Mind 
occurred in the spring of 2003. The 
results were encouraging, but the 
test occurred in only one classroom 
and the inventors of the approach 
were able to directly engage with 
students in the classroom (Weber, 
2003). Although this helped the 
team move forward, the team also 
realized that a product that relies on 
the presence of its inventors during 
implementation cannot scale. 


In 2005-2006, teachers implemented 

the program with less involvement 

of the inventors, but during the first 
year of the program's expansion, support to teachers was 
minimal. Teachers were not taught the curriculum’s peda- 
gogy, so there was no context to guide the content of their 
small group sessions. New classroom practices were needed 
to support this new environment, but no best practices 
were available. Interfaces and reporting tools were neither 
intuitive nor thoroughly explained. As a result, an external 
evaluation found that the quality of the implementations 
was inconsistent and overall low (Weber, 2006). 


To address these issues, Reasoning Mind began to design 
professional development and implementation support in 
the 2006-2007 academic year. A cross-functional team was 
assembled to design the processes and supports that would 
be provided to teachers. The team included Reasoning 
Mind's CEO, Director of Implementation, Principal Analyst, 
content specialists, and experienced Reasoning Mind 
partners. 


The team had considerable data at its disposal, and they 
used the system metrics, classrooms observations, evalu- 
ator reports, and teacher focus groups to target the most 
important implementation issues. Later, when the designs 
below were tested in the field, new data was collected and 
analyzed to drive its evolution. 
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«9 The Implementation Coordinator Role 


The cornerstone of Reasoning Mind's design for implemen- 
tation has been the Implementation Coordinator role. At first, 
mplementation Coordinators were hired to provide train- 
ings and guidance throughout the school year, and their role 
could be considered to be the same as a trainer or profes- 
sional development leader. To anchor this effort, Reasoning 

ind distilled the best practices common in its implementa- 
tions and the recommendations of its expert teachers into a 
rubric used by teachers, administrators, and Implementation 
Coordinators. Therefore, expert modeling drove the direction 
of implementation design just as Instruction Modeling 
shaped the Reasoning Mind curriculum. 


Initially, the role of Implementation Coordinator was similar 
to the role of providing PD. However, the difference in the 
title for the role came to have important meaning. Whereas 
the mission of professional development is usually to 
improve a teachers’ knowledge or skill in particular way, the 
mission of an Implementation Coordinator is to achieve 

a defined quality of implementation. Unlike professional 
development leaders, Implementation Coordinators (ICs) do 
not only deliver PD. Rather, they do what it takes to achieve 
the desired quality of implementation in each classroom. 


For example, ICs initially focused just on teachers. This kept 
classroom practices as a local effort rather than a district 
initiative, and it alienated administrators who lacked the 
knowledge to actively monitor their implementations. The 
team saw this as a problem to be resolved, and expected 
that by engaging administrators, ICs could create higher 


Reasoning Mind 


Solving for every variable 


DASHBOARD 


“G 


FIGURE 4. Part of the Reasoning Mind Community site. 


COLLABORATE 
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PD & RESOURCES 


accountability to implementation fidelity and simultaneously 
lower the cost of implementation per student. Hence ina 
refined implementation approach, ICs now also work with 
school administrators. Likewise, ICs have responsibility to 
communicate needed system improvements back to the 
Reasoning Mind team. Further, ICs work as part of the whole 
team to find solutions to difficult school-specific challenges, 
which may have to do with how instruction is scheduled 

at a school or how students are assigned to teachers or 
classrooms. Thus ICs now broadly coordinate the approach to 
achieving high quality implementation. 


Another difference is how ICs allocate their time and 
attention. In conventional professional development (PD), 
trainers spend roughly equal amounts of time with each 
teacher. ICs, however, have the freedom to spend more time 
with teachers who need more support to improve their 
implementation. Likewise, the program of support can be 
customized to work with the characteristics of a particular 
teacher or school. 


Tools for Coordinating Improvement 


As the number of schools using Reasoning Mind expanded, 
the organization realized that it could not afford to be solely 
responsible for improvement. Instead, Reasoning Mind 
took the approach of engaging teachers and administrators 
as partners in improving implementation. To aid adminis- 
trators and teachers in overseeing their implementations, 
Reasoning Mind created a website called the Reasoning 
Mind Community (see Figure 4). 


Community 


HELP & SUPPORT IDEAS 
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Key elements of the site include: 


- DASHBOARD The community dashboard integrates 


the Reasoning Mind observation application can have 

implementation quality automatically analyzed and 

recommendations to support instruction immediately 

prepared. 

- PD &RESOURCES After reviewing the dashboard’s 
recommendations, the PD & Resources tab enables 


partners to download or sign up for the help they need. 
For convenience, professional development is delivered 
in person, remotely, and online. Its courses are designed 


in collaboration with the curriculum team to ensure 


fidelity to the program's pedagogy and include a variety 


of best practice workshops, pedagogical trainings, and 
curriculum tutorials. 

- COLLABORATE This moderated forum enables part- 
ners to interact with one another over targeted topics. 
Partners use this opportunity to exchange ideas and 


connect with a large network that fosters their growth. 


This peer support creates a culture around a common 
experience of using Reasoning Mind. 


- HELP & SUPPORT For technical questions and help, 


student metrics with recommendations generated from 
observation data. Therefore, an administrator who uses 


partners can download guides or submit questions to the 


Partner Support Team through email and instant chat. 


- IDEAS Teachers and administrators use this space to sub- 


mit ideas for technical improvements, new functionality, 
and classroom resources. Teachers are also able to vote 
on ideas of others that would improve their experience. 


By connecting partners to resources and each other, the 
design team created a Reasoning Mind Community that 
could serve as an integrated hub for administrator and 
teacher support. Related research shows that without this 
cultural support, educators are likely to revert to scripts of 


Hiebert, 2009). 


Defining Guidelines and Practices 


Teachers require time to achieve 


habit rather than adopt new pedagogical methods (Stigler & 


difficulty in the first semester with Reasoning Mind because 
it required so many changes from their traditional teaching 
style. However, by the end of the second semester, teachers 
were more positive, as we will discuss in further detail. 


To support teachers’ developmental trajectory, the 
Reasoning Mind team responsible for designing for im- 
plementation developed and refined clear guidelines and 
practices to drive its professional development, support, 
and resources. These guidelines and practices underwent 
multiple iterations over time. 


The first iteration was an implementation rubric with three 
levels of implementation quality: established, proficient, and 
advanced. The expectation was that an individual teacher 
could have different quality ratings for different areas of their 
implementation. For example, a teacher could be advanced 
in terms of incentives, but only established in terms of da- 
ta-driven decision making. From 2010 to 2014, ICs used this 
rubric to observe teachers. Its one-page, color-coded format 
was important to ease of use by ICs (see Appendix A). 


The design of the implementation rubric evolved as more 
was learned from the implementations it served. At first, the 
implementation rubric was a distillation of effective teacher 
practices. In addition to supporting teachers with this rubric, 
mplementation Coordinators observed classrooms to 
inform their recommendations. As a result, a massive amount 
of Implementation data was collected. 


Evaluations also contributed to the improvement of the 
rubric. For example, analysis of classroom observation and 
student performance data indicated that higher levels of 
implementation fidelity were correlated to stronger student 
outcomes in Reasoning Mind. This was encouraging, as it 
confirmed that many of the practices identified may support 
student success. The graph in Figure 5 shows the correlation 
between implementation fidelity and student problem 
solving accuracy (percentage correct) for the most difficult 
problems in Reasoning Mind's curriculum, level C problems. 
Similar correlations exist for level A and level B problems, 
which are not as hard. 


high quality implementations. 
For example, the current study in 50 
West Virginia was designed so that 
teachers have an entire school year 
to improve their implementation of 
Reasoning Mind. Student learning 


Passing 


results will be measured in a second 10 = 
school year, with a second cohort of 


38 
30 
; fe 


students. Although this may seem Not Yet Established Established Proficient Advanced 


like a long time, observations in 
West Virginia supported the need 


for along developmental period. For FIGURES. In classrooms with higher quality implementation, students get a higher 
example, some teachers reported a percentage of difficult problems correct. 
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However, after many years of using version of rubrics to 
guide ICs in the work of improving implementations, 
Reasoning Mind came to realize that the rubric-based 
approach had both a key benefit and a key flaw. The key 
benefit was that the rubric recognizes areas in which imple- 
mentation is improving without setting the expectation that 
all classrooms will improve at the same rate or in the same 
ways. The key flaw was that the rubric started to breed an 
orientation to compliance. Some ICs, teachers, and schools 
aimed for high marks on the rubric, while still not achieving 
strong student results. Other ICs, teachers and schools were 
inventing new best practices, but the practices were not 
spreading because they did not fit the rubric. This realization 
led to discarding the rubric format (i.e, the matrix of imple- 
mentation characteristics and implementation levels) and 
re-focusing on the desired quality indicators. 


Achieving managed implementation required a clear defi- 
nition of what high quality implementation looks like, but 
to avoid the issues with the leveled rubrics discussed earlier, 
the team wanted goals at the forefront, with the practices 
as examples of how to attain them. After Reasoning Mind's 
analysts studied the rubric variables that had the greatest 
impact on student performance, the Implementation 

team produced the Goals and Practices document (see 
Appendix B), placing a greater emphasis on the suite of 
mplementation components that made the greatest 
impact. This is reinforced by tailored Reasoning Mind 
reports and a mobile application that translates classroom 
observations into recommendations in the Reasoning Mind 
community. 


« Regional Implementation Managers 


Another important design change was in the management 
of implementations. Initially, |Cs were managed centrally 
from Reasoning Mind's offices in Texas. As Reasoning Mind 
spread to other states, regional differences became notice- 
able. The ways in which schools are managed varies from 
state to state (for example, Texas tends to have very large 
school districts whereas West Virginia has very small ones). 
The culture of how teachers and administrators commu- 
nicate is different. Also, some of the challenges schools 
face are similar within regions, but different across regions. 
In response, Reasoning Mind developed a new role for a 
regional IC manager, who works with a team of regional ICs 
to improve implementation. 


The Implementation Manager (IM) coaches a team of ICs, 
each of whom is developing skills in supporting Reasoning 
Mind's implementations in schools and at a higher county 
level. Together, regional teams discuss emerging challenges 
in their area and seek common best practices and solutions 
that will address the challenges. The IM also coordinates with 
Reasoning Mind's central office to discuss improvements and 
developments to the system, Professional Development and 
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materials, and overall support structures which are needed 
to ensure strong partner relationships and high quality 
classroom practices. 


Within its regional strategy, Reasoning Mind is seeking to 
strike a balance between setting high standards for imple- 
mentation quality everywhere, but also seeking to continue 
to innovate in how high quality implementation is defined 
and achieved. The goal is to capture what's working to 
enable students in every Reasoning Mind site to have a very 
high quality mathematics leaning experience, without over- 
emphasizing compliance to practices that are not universally 
necessary, and while retaining the flexibility to add emerging 
best practices into the mix. 


WV IMPLEMENTATION REFLECTIONS 


In West Virginia, the SRI team has been able to reflect on data 
from a first year of implementation improvement across 23 
schools (in the 2014-15 school year). Reflecting on this data 
helps to inform the design issues discussed earlier and is 
important to SRI’s process of developing reference cases that 
SRI can share with confidence across related implementation 
design challenges. Student learning outcomes, however, are 
not yet available and thus SRI is not yet able to report on the 
efficacy of the product. 


Overall, we noted that all teachers were able to complete 
the required Reasoning Mind initial qualification course 

and began using Reasoning Mind. At the beginning of 

the school year, some teachers were slower to start using 
Reasoning Mind, in part due to technology issues. Some 
teachers complained about the system; they didn't like the 
changes to their teaching practices that were required, such 
as spending less time leading the whole classroom and 
more time working with individuals and small groups. Some 
teachers thought the content was too hard for the students. 
Overall, most teachers immediately engaged with Reasoning 
Mind as their core curriculum for 5" grade mathematics 

but teachers varied in their degree of commitment to the 
program. 


Providing ongoing support to engage teachers with the 
Reasoning Mind curriculum and blended learning classroom 
strategies more fully are major roles of the Implementation 
Coordinators (ICs). Assigning each IC to specific first year 

5'" grade teachers allowed for personalized support in the 
initial training, launching of the program, and then ongoing 
support throughout the academic year. ICs are experts in 
Reasoning Mind's system components, student and teacher 
interface functionality, curriculum, and implementation 

best practices and served as coaches who provided direct 
feedback and guidance, as well as remote support. For first 
year teachers this included: (a) an in-person meeting after 
their training to develop an appropriate plan of action for 
introducing students to the Reasoning Mind program; (b) an 
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in-person, on-site training directly 
after launch in which an IC visited a 30 
teacher during Reasoning Mind time 
to model best practice strategies; 


and (c) monthly observations with 0 
follow-up meetings to discuss best 15 
practices, goals, and growth. 10 


Another role of the IC was to monitor 
system performance data and iden- 
tify next steps for professional de- 
velopment on student and teacher 
actions. First year teachers received 
weekly data reports with reflections 
from ICs to help coach them in 


=f == 


Very Satisfied 


Somewhat Somewhat Very Dissatisfied 
Satisfied Dissatisfied 
Wi Control —— | Treatment 


Reasoning Mind's data analysis and 
tutorial strategies. These reports also 
helped set goals for motivation and 
progress toward curriculum comple- 
tion. For professional development, 
each first year teacher was encouraged to take advantage 

of six two-hour Reasoning Mind workshops, three in the fall 
and three in the spring. ICs helped teachers set goals on 
these workshops, monitored their progress, and suggested 
topics for study. For many teachers, Reasoning Mind was 

the first technology-based core curriculum they had taught, 
and this coaching aspect of the support is crucial for teacher 
commitment and fidelity. Because of this, teachers also had 
access to their IC through phone, email support, and the 
Reasoning Mind Community where they could find resourc- 
es, get technical help on system reports or accounts, and 
collaborate with other teachers across the country. 


math curriculum. 


By the end of the year, key indicators reflected a quality of 
implementation that the aforementioned design features 
were intended to support. On average, schools were using 
Reasoning Mind for close to the target 70 minutes a day. 
System data showed that students on average were solving 
over 2,500 math problems per year in Reasoning Mind, and 
these problems included the desired range of problem diffi- 
culties (levels A, B, and C). Further, although student accuracy 
rates in solving the problems varied by school (and by 
classroom), all classrooms were within the range of expected 
accuracy rates. The system use data also shows that teachers 
in the treatment group were using the performance reports 
provided by Reasoning Mind regularly on a daily basis (over 
5 times a day on average). The gradual movement of the in- 
dicators to appropriate levels re-affirmed our understanding 
that with well-designed support for implementation from 
Cs, all teachers can reach the target quality within a year and 
that some teachers reach these levels sooner than others. 


n addition, teachers in both the Treatment group (using 

Reasoning Mind) and in the Control group (using their usual 
curriculum materials) took an end of year survey. Treatment 
teachers were notably more positive about their curriculum 
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FIGURE 6. Reasoning Mind (Treatment) teachers were more satisfied with thei 


a 


resource. For example, when asked "how satisfied were you 
with your curriculum this year’, 26% of treatment teachers 
responded as "very satisfied” whereas less than 3% of control 
teachers chose this option. Six percent of control teachers 
expressed that there were "very dissatisfied” with their 
curriculum, while no treatment teachers responded so (see 
Figure 6). 


Teachers’ positive response to Reasoning Mind appears to be 
related to the features of implementation cited in this article. 
For example, 88% of teachers said the Genie character and 
Genie rules were somewhat or very useful. In addition, 97% 
of teachers were completely satisfied with implementation 
support; the remaining teachers were not dissatisfied with 
implementation support, but described support as a "minor" 
issue. The importance of Implementation Coordinators in 
change management also came through in the teachers’ free 
responses: 


"(IC] was always very helpful and supportive. She was quick 
with responses and very reassuring during classroom visits. 
She made our transition to using RM very easy.” 


"(IC] is always helpful in all communication and feedback 
that | need to be successful. She makes me feel at ease and 
is quick to respond when | have questions.” 


"(IC] was great to answer all questions that | had. She 
always responded to emails within a day and was always 
able to be reached. Her PD was also very useful. My students 
enjoyed seeing her when she came to my class.” 


"[IC] was very dependable and helpful anytime | needed her. 
| contacted her several times with questions or concerns 
that | had. She even came to my class to model a lesson 
that | did not understand” 
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On the basis of this data, Reasoning Mind plans to make the 
implementation design tested in West Virginia its standard 
operating procedure for each new region as it expands. 
Because the data relates the specifics of the design for imple- 
mentation to the quality of implementation observed, SRI 
sees the data as providing empirical support for this design 
for implementation. Empirical support gives SRI confidence 
to use this approach to implementation challenges as a 
reference design in its ongoing work with providers of digital 
instructional materials for other curricular topics. 


CONCLUSION 


The Reasoning Mind design case is a case of a nonprofit 
organization focused on a comprehensive digital curric- 
ulum that observed a need to address implementation 
quality early on, and made steady improvements thereafter. 
Throughout the same time period, Reasoning Mind contin- 
ued to improve its product, for example, adjusting the math 
content to fit new standards, improving the algorithms used 
to assign problems to students, adjusting the behavior of the 
Genie character, etc. Improvement of implementation was a 
distinctive layer of work, with its own challenges and its own 
designs. This layer of design for implementation became 
increasingly important to Reasoning Mind as the use of the 
curriculum reached a larger scale. 


SRI frequently works in a design consulting or evaluation role 
with a range of educational technology materials that are 
in the process of going from smaller scale to broader use. In 
the transition to broader use, challenges of supporting high 
quality implementation by teachers often rise to the sur- 
face—and cannot be addressed merely by design processes 
related to curriculum or user experience. Implementation 
has its own challenges and needs its own design processes, 
as even high quality instructional designs can fail to yield 
intended effects if implementation challenges are not 
addressed. 


One set of challenges addressed in this design case 
was in how to work with school staff on improvement 
issues, as a high quality implementation takes time to 
achieve. Reasoning Mind distinctively designed the role of 
mplementation Coordinator, which is different from the tra- 
ditional role of Professional Development Leader. Reasoning 
ind also realized a need to coordinate not only with 
teachers, but also with school administrative leaders. Later, 
Reasoning Mind shifted to a regional approach, defining the 
role of Regional Implementation Manager, to coordinate 
implementation work across a region. 


Another set of challenges was in defining tools for this work. 
Reasoning Mind developed a community tool to coordinate 
the work of its staff with school staff as they engage in im- 

provement work. Reasoning Mind also developed and used 
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Implementation Rubrics to guide the improvement work. 
These went through multiple iterations of use and improve- 
ment, but ultimately were supplanted by an Implementation 
Goals and Practice document. Reasoning Mind had noted 
that the prior rubric was leading (in some instances) to a 
compliance culture that was diverging from its goals. The 
newer document was built on previous rounds of evaluation 
research to identify the variables which were most closely 
linked to student learning, while leaving more freedom in 
how teachers and schools achieved implementations to 
reach targets. 


A third set of challenges has been in balancing between a 
desire for standardization and recognition of the value of 
continuous improvement. Early in the history of Reasoning 
ind, implementation was not standardized at all, and this 
led to low overall results. In response, new roles and tools (as 
described earlier) were standardized. But Reasoning Mind 
also learned that standardizing the wrong things could lead 
to optimization of variables which don’t matter and inability 
to share emerging best practices which were different but 
also highly successful. Over time, Reasoning Mind has come 
to realize that its key value is the quality of each student's 
mathematical learning experience, and that it should focus 
on those variables which contribute most directly to this 
value, while being less prescriptive about implementation 
variables which are less important to this value. 


Overall, this design case revealed how we addressed these 
challenges through the intertwined evolution of these three 
elements: (a) the definition of key roles and processes, (b) 
the design of tools and structures to support those roles 
and processes, and (c) the clarification of the values of the or- 
ganization in relationship to the specification of its approach 
to implementation improvement. Going forward, Reasoning 
Mind plans to continue using and improving its design for 
implementation as it expands its work to other regions of 
the country. SRI plans to incorporate this design case as an 
exemplar when it works with the providers of other instruc- 
tional materials that are going to scale, and thus need to 
address challenges associated with supporting high quality 
implementation of digital materials across large numbers of 
classrooms. 
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CATEGORY 


DATA DRIVEN 


DECISIONS 


LESSON 
PLANNING 


INSTRUCTIONAL 
METHODS 


LEARNING 
MODES 


TEACHER 
ENGAGEMENT 


PROCEDURES 


INCENTIVE 
SYSTEMS 


NOTEBOOKS 


INDEPENDENT 
LEARNING 


STUDENT 
ENGAGEMENT 


APPENDIX A 


Reasoning Mind’s Implementation Rubric 


ESTABLISHED 


Teacher uses the home screen or 
reports to make instructional 
decisions. 


Teacher starts the class with a 
structured agenda that includes 
intervention time. 


Teacher regularly conducts 
interventions during each class. 


Class is generally logged into 
Reasoning Mind during sched- 
uled time and regularly uses 
Review Mode or Wall of Mastery, 
as evidenced by metrics. 


Teacher generally is directly 
engaged with students during 
class time. 


Teacher has procedural expecta- 
tions in place and intervenes 
when necessary. 


Teacher has goals and rewards 
based on class or individual 
performance, and most students 
can identify them. 


Students have organized 
notebooks that appropriately 
include theory notes and student 
solutions. 


Students know independent 
learning strategies, including the 
Genie’s Rules, and are reminded 
to use them. 


In general, students are directly 
engaged in learning during class 
time, and misbehaviors are often 
redirected. 


REASONING MIND: 
Implementation Rubric 


PROFICIENT 


Teacher uses metric analysis to 
make instructional decisions. 


Teacher uses metric analysis to 
start the class with specific 
students chosen for intervention 
time. 


Teacher uses a variety of 
intervention methods to address 
student struggles. 


Class is logged into Reasoning 
Mind for a high percentage of 
scheduled time and frequently 
uses Review Mode or Wall of 
Mastery, as evidenced by metrics. 


Teacher is directly engaged with 
students for a high percentage of 
class time. 


Teacher has procedures in place 
that run smoothly and require 
little intervention. 


Teacher has goals and rewards 
both for class and individual 
performance; the teacher 
reinforces these goals during each 
class. 


Students’ notebooks include 
additional learning aids, such as 
corrected solutions, accuracy 
tracking, and vocabulary. 


Students use independent 
learning strategies; these 
strategies are reinforced during 
each class. 


Students are directly engaged in 

learning for a high percentage of 
class time, and misbehaviors are 

nearly always redirected. 


ADVANCED 


Teacher uses metric and problem 
analysis to make instructional 
decisions. 


Teacher starts the class with 
specific intervention plans for the 
chosen students, such as key 
problems or additional materials. 


Teacher uses methods to 
challenge high-achieving 
students and anticipates potential 
problems through pre-teaching. 


Class is logged into Reasoning 
Mind for nearly all of scheduled 
time and frequently uses Review 
Mode and Wall of Mastery, as 
evidenced by metrics. 


Teacher is directly engaged with 
students for nearly all of class 
time. 


Teacher has procedures in place 
that run efficiently and do not 
require facilitation. 


Teacher encourages students to 
take ownership of goals, and 
students are highly motivated by 
them. 


Students use their notebooks to 
answer questions and help with 
problem solving. 


Students adeptly use 
independent learning strategies 
without teacher facilitation. 


Students are directly engaged in 
learning for nearly all of class 
time, without teacher redirection. 
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APPENDIX B 


Reasoning Mind’s Implementation Goals and Practices Document 


Reasoning 


Implementation Goals & Practices 


STUDENT PERFORMANCE 


A-level Accuracy A-level accuracy meets standard for 75% of students. 
B-level Accuracy B-level accuracy meets standard for 75% of students. 
C-level Accuracy C-level accuracy meets standard for 75% of students. 
Productivity Productivity meets standard for 90% of students. 
Notebooks Students keep organized notebooks to facilitate learning. 
Foundation v Students have: 


e notebooks. 

e organized and legible notes. 

e consistent theory notes. 

e consistent full solutions. 

Students use their notes to solve problems. 

Classroom procedures support the effective use of notebooks. 
Incentive systems motivate students to meet notebook expectations. 
Students understand and remember their notes. 


Student notebooks include additional learning aids when appropriate. 
All students achieve an average Notes Tests score of 90% of higher. 


v 
Vv 
v 
v 
Mastery v 
v 


Independent Learning Students appropriately employ independent learning strategies. 


Foundation Independent learning strategies are regularly communicated. 
Students know independent learning strategies and when to use them. 
Students use strategies appropriately to progress independently. 
Students read and understand the theory. 

Students read and understand the Genie’s Solution. 

Classroom procedures support independent learning. 


Incentive systems motivate students to learn independently. 


Students employ personalized learning strategies. 
Students use the Genie’s Solution to correct misunderstanding. 


Mastery 


2 Se SS SS 


Focus Students are focused on learning. 


Foundation vy Students are: 

e on task. 

e redirected when off task. 

e not disrupting others. 

Classroom procedures allow for maximum time on task. 

Incentives systems motivate students to be engaged in their learning. 


Students are intrinsically motivated and on task. 
Students help keep other students focused on learning. 


Mastery 


46/66 
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TEACHER PRACTICES 


Data Driven Decisions Teacher uses data appropriately to make instructional decisions. 


Foundation v Notifications, reports, and activity logs are used to determine: 
which students need coaching on independent learning. 
which students need intervention or enrichment. 

e which objectives need pre-teaching or re-teaching. 

e the root of student misunderstanding. 


Mastery vy Data trends are identified and used as context for instructional decisions. 
vy Student notebooks are used to identify needs and inform coaching. 


Instruction Teacher provides individualized instruction to students. 


Foundation nstruction addresses the needs of struggling students. 

nstruction challenges high-achieving students. 

nstruction reinforces independent learning. 

Misconceptions are quickly addressed. 

ntervention plans are devised before class. 

ntervention plans target specific students, concepts, and skills. 
ndividualized instruction comprises the majority of scheduled class time. 
All diagnoses are remedied. 


Teachers appropriately use Reasoning Mind assignments to support 


£6666 66 66 


earning. 


nstruction addresses anticipated struggles through pre-teaching. 
ntervention methods are selected to best suit the student and content. 
ntervention plans include specific and strategic problem sets. 
nstructional methods reinforce Reasoning Mind’s pedagogy. 

nstruction teaches students to state rules and definitions precisely. 
nstruction teaches students to write full solutions. 

nstruction teaches students to articulate mathematical reasoning. 


PROGRAM ACCESS 


Technology Technology is accessible and meets minimum requirements. 


Mastery 


{666666 


Foundation Technology issues do not block students from using Reasoning Mind. 
Technology accessibility is sufficient to meet minimum time requirements. 
Technology issues are identified and quickly addressed by district IT staff 
in collaboration with Reasoning Mind staff. 


Technology changes are proactively communicated to Reasoning Mind IT. 


<<< 


Scheduling Scheduled time is sufficient and well structured. 


Foundation wv Scheduled time is sufficient to meet the minimum time requirements. 
vy Class periods are sufficiently long for productive Reasoning Mind use. 
Mastery vw Class schedules contain flexibility to make up for missed class periods. 
wv Schedule time is sufficient to exceed the minimum time requirements. 
Time Online Students spend their scheduled time in the program. 
Foundation wv Students spend the scheduled amount of time in Reasoning Mind. 
vy Students spend the vast majority of their online time in learning modes. 
Y Students spend an appropriate amount of time in Review and Wall of 
Mastery. 
Teachers make up missed class periods. 
Mastery wv Students exceed minimum time requirements through additional practice. 
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INVESTMENT 


Teacher Investment Teachers are knowledgeable and invested. 


Foundation Teachers know and pursue the Implementation Goals and Practices. 
Teachers take ownership over the success of Reasoning Mind. 
Teachers are knowledgeable about best practices by participating in 
Reasoning Mind workshops and collaborating with others. 

Teachers are knowledgeable about Reasoning Mind’s curriculum and 
pedagogy through workshops, collaboration, and independent study. 


Teachers provide feedback to Reasoning Mind. 


4 SS 


Teachers provide guidance to other Reasoning Mind teachers. 
Teachers share their Reasoning Mind experiences with others. 
Teachers are active in the Reasoning Mind community. 
Teachers seek out additional professional development. 


Mastery 


C4666) 6 


School Administrator School administrators are knowledgeable and invested. 
Investment 


Foundation vy Administrators reinforce the Implementation Goals and Practices. 
Administrators take ownership over the success of Reasoning Mind. 
Administrators are knowledgeable about best practices through 
administrator workshops and collaborating with others. 

vy Administrators ensure a timely launch of their implementations by 
coordinating student access and teacher preparation. 
Administrators are proficient with Reasoning Mind Goals and Practices 
reports and monitor the progress of their implementations. 
Administrators conduct regular classroom walkthroughs to provide 
informed teacher feedback. 

Administrators coordinate their support with Reasoning Mind staff. 
Administrators quickly address issues that inhibit student learning. 
Administrators provide feedback to Reasoning Mind. 


446 


< 


< 


Administrators provide guidance to other Reasoning Mind educators. 
Administrators share their Reasoning Mind experiences with others. 
Administrators are active in the Reasoning Mind community. 
Administrators seek out additional professional development. 


Mastery 


4666) 666 


District Administrator District administrators are knowledgeable and invested. 
Investment 


Foundation Administrators reinforce the Implementation Goals and Practices. 

Administrators take ownership over the success of Reasoning Mind. 

Administrators are knowledgeable about Reasoning Mind and how 

to implement it effectively. 

¥ Administrators are proficient with Reasoning Mind Goals and Practices 
reports and monitor the progress of their implementations. 

vy Administrators coordinate their support with Reasoning Mind staff. 

vy Administrators quickly address issues that inhibit student learning. 

v Administrators facilitate the enrollment process, the launch of Reasoning 

Mind, its implementation, and research efforts. 

Administrators provide feedback to Reasoning Mind. 


KS 


Administrators provide guidance to other Reasoning Mind educators. 
Administrators share their Reasoning Mind experiences with others. 
Administrators are active in the Reasoning Mind community. 
Administrators attend professional development with staff. 


Mastery 


4666) 6 
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